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Overview 


1.  Motivation 
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•  Actor-oriented  Design  for  State  Space  Dynamics  and 
Measurements 
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Motivation 


ML  technology  in  programming  languages: 

•  MATLAB,  Python,  Octave,  Julia,  R  ... 

And  in  the  form  of  toolkits: 

•  GMTK,  Stream  Lab,  SHOGUN,  Weka,... 

The  state-of-the-art  tools  traditionally  interact  with  data  and 
present  no  native  way  of  incorporating  system  aspects 

Goal:  to  make  the  ML  aspects  a  native  part  of  the  system  design  by 

•  Exploiting  component-level  interactions  in  the  swarm 

•  Restoring  the  system  level  roots  of  machine  learning 
methodologies  by  providing  the  right  interfaces  between 
machine  learning  tools  and  CPS  design  aspects. 
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Motivation 


We  present  an  actor-oriented  machine  learning  toolkit 
that  focuses  on 

•  Applications  of  ML  Algorithms  to  streaming  data 

•  Enabling  ML  techniques  to  be  natively  integrated  into 
system  design 

•  Context-aware  parameterization  of  a  rich  set  of  ML 
algorithms 

•  Library  of  easy-to-use  tools  for  developers  who  are  not 
ML  experts 

•  Enhancing  programmability  of  swarmlets 
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Inference  for  Streaming  Data 


Kalman  Filter 
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xq  ~  N( 0,  Pj) 
xt  =  Ftxt_i  +  Btut  +  w  t 
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Wi  ~  N( 0,  Qt),  Vi  ~  N( 0,  Rt) 
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X.  state  (  hidden) 


measurement 

model 


t  observations 
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The  Machine  Learning  Toolkit  in 
Ptolemy  II 


MachineLearning 


▼  l_  HMM 

HMMGaussianEstimator 
HMMExponentialEstimator 
HMMMultinomialEstimator 
El  HMMGaussianClassifier 
El  HMMExponentialClassifier 
O  HMMMultinomialClassifier 

▼  ParticleFilter 
El]  ParticleFilter 

FsH  ParticleFilterRange 

El]  Collaborative  Range  Particle  Filter 

▼  StateSpaceModel 

MeasurementModel 
J3l  StateSpaceModel 
El]  ParticleFilterSSM 
Fvl  PredictorSSM 

►  ZZ:  Mail 

►  L  Metroll 

►  OpenModelica 

▼  l _ Optimization 

l&l  CompositeOptimizer 
[213  TrajectoryOptimizer 


Machine  Learning: 

1.  Hidden  Markov  Models  (HMM) 

2.  Gaussian  Mixture  Models  (GMM) 

•  Parameter  Estimation 

•  Classification 
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Q 


The  Machine  Learning  Toolkit  in 
Ptolemy  II 


MachineLearning 


▼  l_  HMM 

HMMGaussianEstimator 
HMMExponentialEstimator 
HMMMultinomialEstimator 
El  HMMGaussianClassifier 
El  HMMExponentialClassifier 
O  HMMMultinomialClassifier 

▼  ParticleFilter 
El]  ParticleFilter 

FsH  ParticleFilterRange 

El]  Collaborative  Range  Particle  Filter 

▼  StateSpaceModel 

MeasurementModel 
J3l  StateSpaceModel 
El]  ParticleFilterSSM 
Fvl  PredictorSSM 

►  ZZ:  Mail 

►  L  Metroll 

►  OpenModelica 

▼  l _ Optimization 

l&l  CompositeOptimizer 
m  TrajectoryOptimizer 


State  Estimation: 

•  Particle  Filtering 


ParticleFilter 


partideOutput 


stateEstimate 


3  0  0 


Edit  parameters  for  ParticleFi Iter2 


L 


2000 


50 


stateVariableNames:  fx"tV 
bootstrap: 

lowVarianceSampler: 
particleCount: 
outputPartideCount: 
processNoise: 
prior: 
x_update: 
y_update: 
zl: 
z2: 


multivariateCaussian({0.0,0.0},[5.0,  0.0;  0.0,  5.0]) 


{random()*200-  100,random()"200- 100} 


sqrt((rl(0)-x)A2+(rl(l)-y)A2) 


sqrt((r2(0)-x)A2-F(r2(l)-y)A2) 


measurementNoise:  gaussian(0.0,5.0) 


Preferences  Defaults 


Remove 


D  C 


Add 


Commit 
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The  Machine  Learning  Toolkit  in 
Ptolemy  II 


MachineLearning 


▼  l_  HMM 

HMMGaussianEstimator 
HMMExponentialEstimator 
HMMMultinomialEstimator 
El  HMMGaussianClassifier 
El  HMMExponentialClassifier 
O  HMMMultinomialClassifier 

▼  ParticleFilter 
El]  ParticleFilter 

FsH  ParticleFilterRange 

El]  Collaborative  Range  Particle  Filter 

▼  StateSpaceModel 

MeasurementModel 
J3l  StateSpaceModel 
El]  ParticleFilterSSM 
Fvl  PredictorSSM 
Mail 

►  L  Metroll 

►  OpenModelica 

▼  l _ Optimization 

l&l  CompositeOptimizer 
m  TrajectoryOptimizer 


Optimization: 

•  CompositeOptimizer:  An  actor- 
oriented  gradient-descent  solver 


CompositeOptimizer 


0  O  O  Edit  parameters  for  CompositeOptimizer 


mode: 

MIN 

M 

maxEvaluations: 

I  MIN 

dimensionOfOptimizationSpace: 

]  MAX 

numberOfConstraints:  1 


timeHorizon:  l 


OptimizerComposite  has  no  parameters. 


Cancel  Help  [  Preferences  )  Defaults  Remove  Add  Commit 
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Application:  Swarmlets  for 
Cooperative  Robot  Control 


Problem  Definition:  A  team  of  robots,  tracking/pursuing  a 
target. 

Model:  State  Space  Model  of  target  dynamics 

Observations:  Robot  sensor  measurements  (generally 
nonlinear  functions  of  target  position  +  noise) 

Tasks: 

•  Target  State  Estimation 

•  Robot  Path  Planning:  Multiple  Objectives 

Collision/Obstacle  Avoidance,  Pursuit,  SLAM,  Fast  Localization, 
Minimal  Uncertainty, ... 
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Cooperative  Robot  Control : 
Challenges 


•  Cooperation  between  robots 

•  Complex  measurement/noise  models 

•  Range  Measurements  ( e.g.,  RSSI) 

•  Bearings  Measurement  (e.g..  Cameras) 


•  Nonlinear  robot  dynamics 

•  Unknown  Environment 


intruderl 


Robot4 

O 


Robot2 

o 


Robot3 

O 


Robot 
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Cooperative  Robot  Localization: 
State  Space  Models 


et  = 


Xf 

Vt 


Target  state  (  position  ) 


Xo  ~  Unif  orm([— 100, 100]) 
yo  ~  Unif  orm([— 100, 100]) 


= 


z  1 

Z2 


Range  Measurements 


Robot4 

O 


Robot2 

O 


intruderl 


Robot3 

o 


Robot 

O 


zit  —  ||  fit  ~  @t  ||  +  i  —  1,2  Measurement  model 
(jjt  ~  M{ 0,  a2),  a2  =  5.0 

'5.0 

’  0.0 


0.0 

5.0 


)  Target  state  dynamics 


$t+ i  —  0t  +  Vti  vt  ~  M{ 


0 

0 
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Algorithm  Workflow 


1. 

2. 


Robots  make  independent  range 
measurements 

A  centralized  (or  local) 
cooperative  state  estimation 
algorithm  estimates  target 
position  given  measurements 

Robot  trajectories  are  optimized 
w.r.t.  some  objective  function 
based  on  the  estimated  target 
position 

Robots  move  according  to  the 
planned  path 


Robot  1 
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Target  State  Estimation 


X0  ~  7Tx(*o) 
Zt\xt  ~  g(xt,ut,t) 
xt+l\xt  ~  f(xt,Ut,t) 


n 


0 


state  (  hidden) 


measurement 

model 


observations 


•  Given  zt  t=l,...T :  noisy  measurements  of  a  target  state  xt, 

•  Estimate  p(xT  |  z1;T):  Posterior  density  of  the  target  state 

N 

p(xt)  =p(xt\z1:t)  =  ^ ~2wlt5(xt  -x\) 

i—  1 

•  Particle  filtering  is  a  popular  Bayesian  Filtering  technique  to 
solve  this  problem:  Provides  a  density  estimate  of  xT  as  a 
particle  set 

TerraSwarm  Research  Center  11/17/14  1 3 


The  Particle  Filter 


•  Introducing  the  particle  filter: 

-  Sequential  Monte  Carlo  methods  as  a  general 
family 

—  A  Bayesian  filter  that  performs  maximum- 
likelihood  state  estimation  for  state-space  models 
with 

•  nonlinear  dynamics  and  non-Gaussian  noise,  in  the 
general  case 

•  A  stochastic  (and  often  better  performing)  alternative 
of  the  Kalman  filter  (  which  is  only  optimal  for  the 
linear  Gaussian  case) 
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Particle  Filter:  Operation 


Establish  a  prior  belief  of  the  state,  represented  as  a  set  of  particles 

Each  particle  is  a  candidate  "state",  which  is  the  intruder  position  in  this  particular  application 
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Particle  Filter:  Operation 


Make  a  measurement 
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Particle  Filter:  Assigning  Weights 


Assign  weights  to  each  particle  according  to  how  well  it  explains  the 
measurement  (  subject  to  a  measurement  model  and  noise  specification) 


17 


o 


Particle  Filter:  Assigning  Weights 


The  particle  weights  (  under  Gaussian  noise)  would  look  like  the  following: 


x  10  J 
1.4 


a  0.8 

.c 

CD 

|  0.6 


py 


px 


Terre _ 
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Particle  Filter:  Resampling 


•  The  resulting  set  of  particles  would  look  like: 
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Particle  Filter:  Propagation 


•  Propagate  resulting  particles  according  to  dynamics  model 


Range-Only  Stationary  Target  Localization 


particles  • 
robot  position  • 
target  position  • 
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Particle  Filtering  with  Range  Sensors 


Collaborative  Particle  Output 


particles  • 
target 
robotl 
robot2  ■ 
state  w 
estimate 


-20  -15  -10  -5  0  5  10  15  20 
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Two-Observer  Particle  Filter 


Measurement 

Input 


0  o  o 


Edit  parameters  for  ParticleFilter2 


Staite  Space_ 
Model 


Preferences 


particleCount: 

outputParticleCount: 

processNoise: 

prior: 

x_update: 

y_update: 

zl: 

z2: 

measurementNoise: 

Defaults 


2000 


50 


multivariateGaussian({0.0,0.0},[5.0,  0.0;  0.0,  5.0]) 


{random0*200-  100,random0*200- 100} 


sqrt((rl(0)-x)A2+(rl(l)-Y)A2) 


sqrt((r2(0)-x)A2+(r2(l)-y)A2) 


gaussian(0. 0,5.0) 


stateVariableNames:  fxVv"} 

i  ^  i 

bootstrap: 

lowVarianceSampler:  ^ 

0*  = 

i  i 

rH-  cs» 

1 _ 1 

Remove 


Add 


Commit 


xq  ~  Unif  orm([— 100, 100]) 
yo  ~  Unif  orm([— 100, 100]) 
zi 


z  t 


Z2 


Zit  =  bit  -  0t II  +U)u  *  =  1,2 
u >t  ~  7V"(0,  o’2),  a2  —  5.0 

0t+ i  =0t  +  vt,  vt  ~  A/'( 


O' 

o 

o 

o 

iO 

0 

? 

0.0  5.0 
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Path  Planning 


•  One  candidate  metric  to  be  used  for  online  trajectory 
optimization:  Information  based  methods:  Mutual 
Information 

-  A  particle  set  is  a  good  probabilistic  measure  of  the 
uncertainty  in  a  state  variable 

-  Size  of  particle  set  can  be  used  to  tune  approximation 
bounds 

•  Optimization  Goal:  Maximize  Mutual  Information 
between  measurements  and  particle  set: 

-  Locate  intruder  as  precisely  as  possible,  with  fewest  steps 

•  Can  equivalently  be  formulated  as:  Minimize 
uncertainty  in  estimated  intruder  location 
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An  Actor-oriented  Optimizer 


Consider  the  general  constrained 
optimization  problem  of  the  form: 


minimize  f(x,  QC 

xGlR™ 

subject  to  g(x,  Q)  >  0 


CompositeOptimizer 
x* 


Currently  supports:  COBYLA, 
a  gradient-descent  constrained 
optimization  solver 
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Cost  Functions  for  Path  Planning: 
Mutual  Information 


Optimization  Goal:  Maximize  Mutual 
Information  between  future 
measurements  and  predicted  particle 
set: 

•  Locate  intruder  as  precisely  as 
possible,  with  fewest  steps 
This  can  equivalently  be  formulated  as: 
Minimizing  the  uncertainty  in  estimated 
intruder  location.  One-step  optimal 
trajectories: 

CompositeOptimizer 


s.t 


ut*  =  argmax  I{zt+l,xt+i) 

U  t 


<  V. 


_  y  max  i 


SDFDirector 


•  speedLimit:  SpeedLimit 


particles 


Figure  :  The  system-level  optimization  problem  for  the  WSN  example 
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Cooperative  Target  Localization: 
Models 
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Demo:  Ml  Maximization 
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Trajectories  and  the  Particle  Filter  Density  Estimate 
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Mean-Square  Localization  Error 
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Demo:  Direct  Pursuit 


T rajectories  and  the  Particle  Filter  Density  Estimate 
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Demo:  Hybrid  Approach  - 1  Follower 
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Bridging  Actor-Oriented  Modeling 
and  ML  Algorithms 


•  Goal:  ML  Algorithms  that  are  aware  of  the 
system  models 

•  Methodology:  Implement  measurement  models 
and  system  dynamics  as  decorator  actors  in  the 
system  model 

-  Easy  to  share,  consistent  models  of  underlying 
system  models 

-  Scalable  and  unambiguous  ML  algorithm  design  for 
non-experts 
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Shared  State  Space  Models  for 
Model  Predictive  Control 


DE  Director 


Simulation  Parameter 


Size  of  Robot  Team,M 
Search  space 

Sensor  measurement  noise 
Maximum  Robot  Speed 
Iteration  Frequency 


Target  Dynamics 

Prior  Belief  on  Target  Position 
(ttx) 


Value 


200x200  units 
vt  ~  M" (0,5.0) 
20  units/s 
10  Hz 


Circular  Motion  with  u  =  7t/5 
Uniform  over  search  space 


Initial  Robot  Configuration 

R|t=o  = 

~Rq~ 

Ri 

r2 

Lr3J 

= 

x  y 

-105.0  -5.0- 

100.0  5.0 

105.0  5.0 

100.0  -5.0 

Initial  Intruder  Position 

I|t=o  =  1—50  50] 

Robot  Control  Inputs 

Robot  Dynamics 

u[l)  =  [vx  vy]  ,  i  e  {0,  1,  2,  3} 

Rit+i  =  RH  +  u^At,  i  e 
{0,1,  2,  3} 

Controller 


g}a 


r 

a 

t 
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Particle  Filter 


Particles 


□Ja 


1-Step  Predictor 


Future  Particle 


1 , * 


XYScope2 


5 -Step  Predictor 


Future  Particle: 

I—1 J 
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Measurement  Models  and 
Dynamics  as  Decorators 


DE  Director  oVtarget:  1  o  Nout:  BOO  o  Nparticles:  2000  Author:  I Ige  Akkaya 

-  o  target_pos:  {x  =  0.178875011368,  y  =  0.4636026512614} 

o  robot_pos:  {x  =  9.5335620567504,  y  =  -2.4478026950311} 

TargetDynamics  o  robot_pos2:  {x  =  57.5521973049689,  y  =  9.5335620567504} 


y  =  g(x,  u,  t)  +  v 
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Measurement  Models  and 
Dynamics  as  Decorators 


DE  Director 

•  Vtarget:  1  •  Nout:  300  •  Nparticles:  2000  Authop  i 

£ 

•  target_pos:  {x  =  0.1 

Controller 


g}a 


stateVariableNames: 

prior: 

processNoise: 

x_update: 

y_update: 


Range  , 
Measurement2 


contraNngut^ 


COryolInput^ 
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{randomO^OO-  100,random02200- 100} 


rnultivariateGaussianftO.O.O.OhfLO.O.^OAl.^]) 


Noisy  Range  Measurement  Model 


StateSpaceModel 


|sqrt((x-  robot pos  .x)A2 +(y-  robot pos  .y)A2) 


►  qK=> 
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[ - 

Rob^ 

►  g}a 

[  ParticleFi Iter 

[  RangeSenso^^  StateSpaceModel  RangeSensorl 

Nparticles 

Nout 

XYScope2 
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Target  Localization:  Adding  a  new 
Sensor 


DE  Director  oVtarget:  1  oNout:300  •  Nparticles:  2000  Author:  I Ige  Akkaya 

-  otarget_pos:  {x  =  0.178875011368,  y  =  0.4636026512614} 

o  robot_pos:  {x  =  9.5335620567504,  y  =  -2.4478026950311} 


TargetDynamics  o  robot_pos2:  {x  =  57.552 1973049689,  y  =  9.5335620567504} 
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Demo:  Prediction 


Range-Only  Stationary  Target  Localization 
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;  service  discovery 


O 


ML  and  Optimization:  Swarmlets 
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Conclusions 


Presented  an  actor-oriented 
machine  learning  toolkit  that  is 
designed  for 

•  ML  and  Optimization 
applications  on  streaming 
data 

•  Enhancing  programmability 

of  swarm  lets 

•  Actor  libraries  for  common 
state-space  dynamics  and 
sensor  models 
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Looking  Ahead 


•  Enhancing  ML  capabilities: 

•  Discrete  Optimization  Solvers 

•  (Mixed)  Integer  Programming 

•  Tool  Integration:  e.g.,  GMTK 

•  Developing  Swarmlets:  Providing  Services  to  TerraSwarm 
Application  Developers 

•  More  case  studies 

•  Anomaly  detection 

•  Multi-sensor  fusion 
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Demos:  Available  in  Ptolemy  II 


Optimization  and  Machine  Learning 

Control  Improvisation 

•  Jazz  Improvisation  http://chess.eecs.berkeley.edu/ptexternal/ 

Optimization 

•  Constrained  Simple  Linear  Regression 

•  Simple  Function  Minimization 

Particle  Filter 

•  Multi  Robot  Intruder  Tracking 

•  Online  Robot  Trajectory  Optimization 

•  Online  Robot  Trajectory  Optimization  -  Distributed  Computation 

•  Open-Loop  Target  Localization  -  Single  Robot 

•  Open-Loop  Target  Localization  -  Two  Robots 

•  Multi-Observer  Particle  Filtering 

•  Particle  Filter  Range 

Probabilistic  Models 

•  Channel  Fault  Model 

•  Communication  Anomaly  Detection  Using  HMM  Estimation 

+  Gaussian  Mixture  Model 

•  Gaussian  Mixture  Model  Parameter  Estimation 

•  Hidden  Markov  Model 

+  Hidden  Markov  Model  Analysis 

•  Discrete-Time  Markov  Chain 
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Thank  You  ! 


Questions? 

Comments? 
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